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DETAILED ACTION 

Response to Arguments 

1 . Applicant's arguments, see Remarks, filed 01/25/2005, with respect to the 
rejection(s) of claim(s) 1,6, 11, and 1 7 under 35 USC 1 03 have been fully considered 
and are persuasive. Therefore, the rejection has been withdrawn. However, upon 
further consideration, a new ground(s) of rejection is made in view of "The 1998 BBN 
BYBLOS Primary System applied to English and Spanish Broadcast News 
Transcription" Spyros et al., 1999, DARPA Broadcast News Workshop (hereinafter 
Spyros) and Sukkar US 5440662 A (hereinafter Sukkar). 

Re Remarks II- A-C (claim 1 pages 12-14), Examiner believes that Naito and 
Shao do not properly teach this limitation and are no longer applicable, specifically " 
creating a gender-independent phoneme model when the difference between the 
compared female phoneme model and the corresponding male phoneme model is less 
than predetermined value". While gender dependence and independence models are 
taught, there exists no mention of its respect gender dependence and independence 
phonemic model generation. Therefore Spyros has been incorporated and Naito and 
Shao withdrawn. 

Re Remarks page 17, "computing accumulated confidence values" are not 
identified by the office action. Examiner agrees and has therefore incorporated Sukkar 
and withdrawn Naito and Wark. 
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Claim Rejections - 35 USC § 103 

2. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subj7ect matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1 -1 6 are rejected under 35 U.S.C. 1 03(a) as being unpatentable over Neti 
et al. US 5953701 A (hereinafter Neti) in view of "The 1998 BBN BYBLOS Primary 
System applied to English and Spanish Broadcast News Transcription" Spyros et al., 
1999, DARPA Broadcast News Workshop (hereinafter Spyros) and further in view of 
Kanevsky et al. US 6529902 (hereinafter Kanevsky). 

Re claims 1 , 6, and 1 1 , Neti teaches a method for generating a speech 
recognition model, the method comprising: 

receiving female speech training data (Abstract, Col. 3 lines 37-49, Col. 4 lines 
10-29, male training data, female training data, gender specific phone state models); 

generating female phoneme models based on the female speech training data 
(Abstract, Col. 3 lines 37-49, Col. 4 lines 10-29, male training data, female training data, 
gender specific phone state models); 

receiving male speech training data (Abstract, Col. 3 lines 37-49, Col. 4 lines 10- 
29, male training data, female training data, gender specific phone state models); 
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generating male phoneme models based on the male speech training data 
(Abstract, Col. 3 lines 37-49, Col. 4 lines 10-29, male training data, female training data, 
gender specific phone state models); 

determining a difference between each female phoneme model and each 
corresponding male phoneme model (Abstract, Col. 3 lines 37-49, Col. 4 lines 10-29, 
aligning data with gender independent data, male training data, female training data, 
gender specific phone state models) 

However, Neti fails to teach phoneme based models and 

creating a gender-independent phoneme model when the difference between the 
compared female phoneme model and the corresponding male phoneme model is less 
than predetermined value 

Spyros teaches a phoneme based gender independent model creation wherein, 
input is segmented into two band-specific episodes using a dual-band gender- 
independent phoneme decoder. Each channel episode is then segmented and gender 
classified in one step with a dual-gender context-dependent word decoder. Further 
Spyros teaches Speaker-Independent (SI), Gender-Dependent (GD) bandspecific 
models are estimated from the training data (Spyros Section 2). 

Further, Spyros teaches segments that contain mixed bands and/or genders. The 
solution is to detect channel changes first, and then detect gender changes within each 
channel, independently. In the 1998 system we used a dual-band gender independent, 
context independent phone-class model for band detection, trained using the following 
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procedure... Spyros teaches training a gender independent (Gl), context independent 
biphone class model, using the above labels (Spyros section 3.4). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to modify the system of Neti to incorporate creating a gender- 
independent phoneme model when the difference between the compared female 
phoneme model and the corresponding male phoneme model is less than 
predetermined value as taught by Spyros to allow for incorporating the use of gender 
dependent and independent phoneme decoder to create a model based on training data 
(Spyros Section 2), wherein the need that exists to model gender differences that are 
not sufficiently modeled by context-dependent variations of Neti can be improved to be 
phoneme based, such as through the use of Spyros in order to handle non-existent 
periods of silence by detecting a difference in a channel followed by a difference in 
gender by first applying a gender dependent model followed by a gender independent 
model when phonemes can not be aligned and seperated into male and female (Spyros 
section 3.4). 

However, Neti in view of Spyros fails to teach 

adding, based on at least one criteria, 

one of the gender-independent phoneme model, OR 

both the female phoneme model and the corresponding male phoneme model to 
the speech recognition model 
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Kanevsky can easily substitute male and female for topics when executing a 
Kullback-Liebler distance method, wherein Kanevsky teaches referring to FIG. 5, which 
illustrates on one-way direction process of separating features belonging to different 
topics and topic identification via a Kullback-Liebler distance method, texts that are 
labeled with different topics are denoted as 501 (e.g., topic 1 ), 502 (e.g., topic 2), 503 
(e.g., topic 3), 504 (e.g., topic N) etc. Textual features can be represented as 
frequencies of words, a combination of two words, a combination of three words etc. On 
these features, one can define metrics that allow computation of a distance between 
different features. For example, if topics T.sub.i give rise to probabilities 
P(w.sub.t.vertline.T.sub.t), where w.sub.t run all words in some vocabulary, then a 
distance between two topics T.sub.i and T.sub.j can be computed as #EQU13##. Using 
Kullback-Liebler distances is consistent with likelihood ratio criteria that are considered 
above, for example, in Equation (6). Similar metrics could be introduced on tokens that 
include T-gram words or combination of tokens, as described above. Other features 
reflecting topics (e.g., key words) can also be used. For every subset of k features, one 
can define a k dimensional vector. Then, for two different k sets, one can define a 
Kullback-Liebler distance using frequencies of these k sets. Using Kullback-Liebler 
distance, one can check which pairs of topics are sufficiently separated from each other. 
Topics that are close in this metric could be combined together. For example, one can 
find that topics related to "LOAN" and "BANKS" are close in this metric, and therefore 
should be combined under a new label (e.g. "FINANCE"). Also, using these metrics, 
one can identify in each topic domain textual feature vectors ("balls") that are sufficiently 
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separated from other "balls" in topic domains. These "balls" are shown in FIG. 5 as 
505, 506, 504, etc. When such "balls" are identified, likelihood ratios as in FIG. 1, are 
computed for tokens from these "balls". (Kanevsky Col. 12 lines 15-56) 

Further, Kanevsky teaches another instance of detecting whether a threshold is 
breached and topic similarity based on training data (Kanevsky Col. 13 lines 7-12 & 
lines 42-45). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to modify the system of Neti in view of Spyros to incorporate 
adding, based on at least one criteria, one of the gender-independent phoneme model, 
or both the female phoneme model and the corresponding male phoneme model to the 
speech recognition model as taught by Kanevsky to allow for the generation of 
combined data models with similar context such as male and female together (e.g. 
LOAN and BANKS) and also isolated data such as explicit male and female data (e.g. 
medical and legal), wherein topics are labeled as a group of phonemes or unigrams 
utilizing a Kullback-Liebler distance, where one can check which pairs of topics are 
sufficiently separated from each other provided a subset of k features, that one can 
define a k dimensional vector allowing computation of a distance between different 
features in the form of a trained group of model (Kanevsky Col. 12 lines 15-56). 
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Re claims 2, 7, and 12, Neti fails to teach the method at least one computer 
readable medium of claim 1 , wherein the at least one criteria comprises a threshold 
value or an upper limit for the total number of phoneme models in the speech 
recognition model. 

Spyros teaches the creation of three models: a triphone within-word Phoneme- 
Tied Mixture (PTM) model with 50 phonetic codebooks, 256 Gaussians per phone, and 
approximately 29K mixture weight vectors associated with the codebooks; a quinphone 
within-word State-Clustered-Tied Mixture (SCTM) model with approximately3500 states, 
64 Gaussians per state, and 30K mixture weight vectors; and a quinphone between- 
word SCTM model with a similar number of parameters (Spyros section 2). 

Additionally, Spyros teaches a phoneme based gender independent model 
creation wherein, input is segmented into two band-specific episodes using a dual-band 
gender-independent phoneme decoder. Each channel episode is then segmented and 
gender classified in one step with a dual-gender context-dependent word decoder. 
Further Spyros teaches Speaker-Independent (SI), Gender-Dependent (GD) 
bandspecific models are estimated from the training data (Spyros Section 2). 

Further, Spyros teaches segments that contain mixed bands and/or genders. The 
solution is to detect channel changes first, and then detect gender changes within each 
channel, independently. In the 1998 system we used a dual-band gender independent, 
context independent phone-class model for band detection, trained using the following 
procedure... Spyros teaches training a gender independent (Gl), context independent 
biphone class model, using the above labels (Spyros section 3.4). 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to modify the system of Neti to incorporate wherein the at least one 
criteria comprises a threshold value or an upper limit for the total number of phoneme 
models in the speech recognition model as taught by Spyros to allow for multiple 
probabilistic models used to differentiate parameters as well as incorporating the use of 
gender dependent and independent phoneme decoder to create a model based on 
training data (Spyros Section 2), wherein the need that exists to model gender 
differences that are not sufficiently modeled by context-dependent variations of Neti can 
be improved to be phoneme based, such as through the use of Spyros in order to 
handle non-existent periods of silence by detecting a difference in a channel followed by 
a difference in gender by first applying a gender dependent model followed by a gender 
independent model when phonemes can not be aligned and seperated into male and 
female (Spyros section 3.4). 

Re claims 3, 8, and 13, Neti fails to teach the method of claim 1 , wherein 
determining the difference includes calculating a Kullback Leibler distance between the 
each female phoneme model and the each corresponding male phoneme model. 

However, Neti fails to teach phoneme based models and 

Spyros teaches a phoneme based gender independent model creation wherein, 
input is segmented into two band-specific episodes using a dual-band gender- 
independent phoneme decoder. Each channel episode is then segmented and gender 
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classified in one step with a dual-gender context-dependent word decoder. Further 
Spyros teaches Speaker-Independent (SI), Gender-Dependent (GD) bandspecific 
models are estimated from the training data (Spyros Section 2). 

Further, Spyros teaches segments that contain mixed bands and/or genders. The 
solution is to detect channel changes first, and then detect gender changes within each 
channel, independently. In the 1998 system we used a dual-band gender independent, 
context independent phone-class model for band detection, trained using the following 
procedure... Spyros teaches training a gender independent (Gl), context independent 
biphone class model, using the above labels (Spyros section 3.4). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to modify the system of Neti to incorporate phoneme based models 
as taught by Spyros to allow for incorporating the use of gender dependent and 
independent phoneme decoder to create a model based on training data (Spyros 
Section 2), wherein the need that exists to model gender differences that are not 
sufficiently modeled by context-dependent variations of Neti can be improved to be 
phoneme based, such as through the use of Spyros in order to handle non-existent 
periods of silence by detecting a difference in a channel followed by a difference in 
gender by first applying a gender dependent model followed by a gender independent 
model when phonemes can not be aligned and seperated into male and female (Spyros 
section 3.4). 
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However, Neti in view of Spyros fails to teach determining the difference includes 
calculating a Kullback Leibler distance 

Kanevsky teaches referring to FIG. 5, which illustrates on one-way direction 
process of separating features belonging to different topics and topic identification via a 
Kullback-Liebler distance method, texts that are labeled with different topics are 
denoted as 501 (e.g., topic 1 ), 502 (e.g., topic 2), 503 (e.g., topic 3), 504 (e.g., topic N) 
etc. Textual features can be represented as frequencies of words, a combination of two 
words, a combination of three words etc. On these features, one can define metrics that 
allow computation of a distance between different features. For example, if topics 
T.sub.i give rise to probabilities P(w.sub.t.vertline.T.sub.t), where w.sub.t run all words 
in some vocabulary, then a distance between two topics T.sub.i and T.sub.j can be 
computed as #EQU1 3##. Using Kullback-Liebler distances is consistent with likelihood 
ratio criteria that are considered above, for example, in Equation (6). Similar metrics 
could be introduced on tokens that include T-gram words or combination of tokens, as 
described above. Other features reflecting topics (e.g., key words) can also be used. 
For every subset of k features, one can define a k dimensional vector. Then, for two 
different k sets, one can define a Kullback-Liebler distance using frequencies of these k 
sets. Using Kullback-Liebler distance, one can check which pairs of topics are 
sufficiently separated from each other. Topics that are close in this metric could be 
combined together. For example, one can find that topics related to "LOAN" and 
"BANKS" are close in this metric, and therefore should be combined under a new label 
(e.g. "FINANCE"). Also, using these metrics, one can identify in each topic domain 
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textual feature vectors ("balls") that are sufficiently separated from other "balls" in topic 
domains. These "balls" are shown in FIG. 5 as 505, 506, 504, etc. When such "balls" 
are identified, likelihood ratios as in FIG. 1 , are computed for tokens from these "balls". 
(Kanevsky Col. 12 lines 15-56) 

Further, Kanevsky teaches another instance of detecting whether a threshold is 
breached and topic similarity based on training data (Kanevsky Col. 13 lines 7-12 & 
lines 42-45). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to modify the system of Neti in view of Spyros to incorporate 
determining the difference includes calculating a Kullback Leibler distance as taught by 
Kanevsky to allow for the generation of combined data models with similar context such 
as male and female together (e.g. LOAN and BANKS) and also isolated data such as 
explicit male and female data (e.g. medical and legal), wherein topics are labeled as a 
group of phonemes or unigrams utilizing a Kullback-Liebler distance, where one can 
check which pairs of topics are sufficiently separated from each other provided a subset 
of k features, that one can define a k dimensional vector allowing computation of a 
distance between different features in the form of a trained group of model (Kanevsky 
Col. 12 lines 15-56). 

Re claims 4, 9, and 14, Neti in view of Spyros fails to teach the method of claim 
3, wherein the difference is a threshold Kullback Leibler distance quantity. 
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Kanevsky teaches the Kullback-Leibler distance (Kanevsky Col. 5, lines 9-11) 
between any two topics is at least h, where h ~s some sufficiently large threshold, also 
Kanevsky teaches (Kanevsky Col. 12, lines 44-47) that while using the Kullback-Leibler 
distance, one can check which pairs of topics are sufficiently separated from each other, 
and that topics that are close in this metric could be combined together). 

Kanevsky also explicitly teaches how a difference is sufficient, such as 
classifying data groups when compared, and also creating independence from 
classificaiton if there is no topic discovered (Kanevsky Col. 5 lines 8-25). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to modify the system of Neti in view of Spyros to incorporate 
whether the model information is insignificant is based on a threshold Kullback Leibler 
distance quantity as taught by Kanevsky to allow for an improved language modeling for 
automatic speech decoding and differentiation between data groups, wherein a 
sufficiently large threshold indicates either separate or combinational probabilities 
(Kanevsky Col. 2, lines 50-52). 

Re claims 5, 1 0, and 1 5, Neti teaches method of claim 1 , wherein the female 
phoneme models, male phoneme models, and gender-independent phoneme models 
are Gaussian mixture models (Neti Col. 3 lines 50-67). 

However, Neti fails to teach gender-independent phoneme models 
Spyros teaches a phoneme based gender independent model creation wherein, 
input is segmented into two band-specific episodes using a dual-band gender- 
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independent phoneme decoder. Each channel episode is then segmented and gender 
classified in one step with a dual-gender context-dependent word decoder. Further 
Spyros teaches Speaker-Independent (SI), Gender-Dependent (GD) bandspecific 
models are estimated from the training data (Spyros Section 2). 

Further, Spyros teaches segments that contain mixed bands and/or genders. The 
solution is to detect channel changes first, and then detect gender changes within each 
channel, independently. In the 1998 system we used a dual-band gender independent, 
context independent phone-class model for band detection, trained using the following 
procedure... Spyros teaches training a gender independent (Gl), context independent 
biphone class model, using the above labels (Spyros section 3.4). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to modify the system of Neti to incorporate gender-independent 
phoneme models as taught by Spyros to allow for incorporating the use of gender 
dependent and independent phoneme decoder to create a model based on training data 
(Spyros Section 2), wherein the need that exists to model gender differences that are 
not sufficiently modeled by context-dependent variations of Neti can be improved to be 
phoneme based, such as through the use of Spyros in order to handle non-existent 
periods of silence by detecting a difference in a channel followed by a difference in 
gender by first applying a gender dependent model followed by a gender independent 
model when phonemes can not be aligned and separated into male and female (Spyros 
section 3.4). 
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4. Claims 17-27 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Neti et al. US 5953701 A (hereinafter Neti) in view of Sukkar US 5440662 A 
(hereinafter Sukkar). 

Re claims 17, 21 , and 24, Neti teaches a system for recognizing speech data 
from an audio stream originating from one of a plurality of data classes ([0094]), each 
data class having a class-dependent phoneme model, the system comprising: 

a computer processor (Col. 6 lines 24-49); 

a receiving module configured to receive a current feature vector of the audio 
stream (Col. 6 lines 24-49); 

a first computing module configured to compute a current best estimates (Col. 3 
lines 50-67) that the current feature vector belongs to one of the plurality of data classes 
(Col. 5 lines 9-21); 

However, Neti fails to teach a second computing module configured to compute 
an accumulated confidence values for each of the plurality of data class that the current 
feature vector belongs to each one of the plurality of data classes, the confidence value 
for each data class of the plurality of data classes based on the current best estimatefor 
the data class and on previous confidence values for the data class, the previous 
confidence values associated with previous feature vectors of the audio stream; 

a weighing module configured to weigh the class-dependent phoneme models 
based on the accumulated confidence values; and 
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a recognizing module configured to recognize the current feature vector (based 
on the weighted class-dependent phoneme models; and 

Sukkar teaches a two-pass classification system and method that post-processes 
HMM scores with additional confidence scores to derive a value that may be applied to 
a threshold on which a keyword verses non-keyword determination may be based. The 
first stage comprises Generalized Probabilistic Descent (GPD) analysis which uses 
feature vectors of the spoken words and the HMM segmentation information (developed 
by the HMM detector during processing) as inputs to develop a first set of confidence 
scores through a linear combination (a weighted sum) of the feature vectors of the 
speech. The second stage comprises a linear discrimination method that combines the 
HMM scores and the confidence scores from the GPD stage with a weighted sum to 
derive a second confidence score. The output of the second stage may then be 
compared to a predetermined threshold to determine whether the spoken word or words 
include a keyword (Sukkar Abstract). 

Further, Sukkar teaches comparisons amongst all scores to produce the highest 
likelihood score to classify a keyword from the words present (Sukkar Col 5 lines 14- 
27). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to modify the system of Neti to incorporate a second computing 
module configured to compute an accumulated confidence values for each of the 
plurality of data class that the current feature vector belongs to each one of the plurality 
of data classes, the confidence value for each data class of the plurality of data classes 
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based on the current best estimate for the data class and on previous confidence values 
for the data class, the previous confidence values associated with previous feature 
vectors of the audio stream; 

a weighing module configured to weigh the class-dependent phoneme models 
based on the accumulated confidence values; and 

a recognizing module configured to recognize the current feature vector (based 
on the weighted class-dependent phoneme models as taught by Sukkar to allow for the 
highest likelihood score selected from a weighted sum of vectors originally obtained 
from speech (Sukkar Col 5 lines 14-27), wherein speech can be properly classified 
based on the best confidence score (Sukkar Abstract), thereby improving the 
probabilistic modeling of Neti to include grouping the highest confidence scores to 
classify speech and test the reliability of that classification. 

Re claims 18, 22, and 25, Neti teaches the method of claim 17, wherein 
computing the current vector probability includes estimating a posteriori class probability 
for the current feature vector (Col. 2 lines 1-8)) 

Re claims 19, 23, and 26, Neti fails to teach the method of claim 17, wherein 
computing the accumulated confidence level further comprising weighing the current 
vector probability more than the previous vector probabilities. 
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Sukkar teaches a two-pass classification system and method that post-processes 
HMM scores with additional confidence scores to derive a value that may be applied to 
a threshold on which a keyword verses non-keyword determination may be based. The 
first stage comprises Generalized Probabilistic Descent (GPD) analysis which uses 
feature vectors of the spoken words and the HMM segmentation information (developed 
by the HMM detector during processing) as inputs to develop a first set of confidence 
scores through a linear combination (a weighted sum) of the feature vectors of the 
speech. The second stage comprises a linear discrimination method that combines the 
HMM scores and the confidence scores from the GPD stage with a weighted sum to 
derive a second confidence score. The output of the second stage may then be 
compared to a predetermined threshold to determine whether the spoken word or words 
include a keyword (Sukkar Abstract). 

Further, Sukkar teaches comparisons amongst all scores to produce the highest 
likelihood score to classify a keyword from the words present (Sukkar Col 5 lines 14- 
27). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to modify the system of Neti to incorporate computing the 
accumulated confidence level further comprising weighing the current vector probability 
more than the previous vector probabilities as taught by Sukkar to allow for the highest 
likelihood score selected from a weighted sum of vectors originally obtained from 
speech (Sukkar Col 5 lines 14-27), wherein speech can be properly classified based on 
the best confidence score (Sukkar Abstract), thereby improving the probabilistic 
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modeling of Neti to include grouping the highest confidence scores to classify speech 
and test the reliability of that classification. 

Re claims 20 and 27, Neti teaches the method of claim 17, further comprising 
determining if another feature vector is available for analysis (Col. 6 lines 24-49). 



Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to MICHAEL C. COLUCCI whose telephone number is 
(571)270-1847. The examiner can normally be reached on 8:30 am - 5:00 pm , Monday 
- Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Richemond Dorvil can be reached on (571)-272-7602. The fax phone 
number for the organization where this application or proceeding is assigned is 571 - 
273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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